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ABSTRACT 

This speech describes the services ideally provided 
by an evaluation department, gives examples of actual services, 
outlines problems that evaluation departments face in delivering 
these services, and discusses the level of funding necessary to 
provide services required by both the community and the U-S. Office 
of Education. According to the report, departments of evaluation 
should, ideally, delineate, obtain, and disseminate information 
useful for making educational decisions. (Author/JF) 



* 



u» 



I 

[ 



C* o 






< V 






THE PURPOSE AND PROBLEMS OF PUBLIC SCHOOL 
DEPARTMENTS OF i EVA LUAT I ON 



*S&: 

I; 

‘A'< 



O 




O 

o 




Presented at 

AERA Symposium on the Use of Federal Funds in Evaluation 
American Educational Research Association 
Annual Meeting, 1972 

by 

Howard 0. Merriman, Ph.D. 

Executive Director 

Department of Evaluation, Research and Planning 
Columbus Public Schools 



w. 



THE PURPOSE AND PROBLEMS OF PUBLIC SCHOOL 
DEPARTMENTS OF EVALUATION 

(The problems they incur, and the necessary level of funding 
to provide the services required by the community and the U.S. 
Office of Education) 



I ntroduction 

This paper first describes the services ideally provided by an evalua- 
tion department, with examples given of actual services, a description of 
the problems which evaluation departments incur in delivering these services, 
and finally, a discussion of, and proposal for the necessary level of fund- 
ing to provide services which are required by both community and U.S. Office 
of Education. 



Services Provided 




The ideal service goal of a department of evaluation can be formulated 

s 

from the Phi Delta Kappa National Commission on Evaluation definition of 
educational evaluation: to delineate, obtain and provide useful information 

for making educational decisions. The institution in which I work, the 
Columbus, Ohio Public Schools, has developed goals for each of its adminis- 
trative divisions. Those which were specified for the division in which the 
evaluation department operates are as follows: 

1. To contribute to the improvement of the program, services, 
operation and administration of the Columbus Public Schools 
who are producing and reporting .factua I information. 

2. To contribute to the improvement of communication between 
and among all individuals and groups in the educational 
community of Columbus, including the Board of Education, 
employees of the Board, pupils, parents, and citizens who 
manifest an interest in school affairs. 

3. To help the Board of Education and school administration 
account to pupils, parents, and the citizens of Columbus 



2 

for the performance of the entire school system and group's 
of pupi Is. 

4. To create, maintain, and improve opportunities for pupils, 
parents, and the citizens of Columbus to have broader and 
more meaningful involvement in school affairs. 

5. To reflect the opinion and attitudes of pupils, parents, 
and the citizens of Columbus about school affairs to the 
Board of Education and school administration. 

6. To reinforce and service the specific evaluation and informa- 
tion needs of all other divisions, departments and schools in 
the Columbus City School District. 

These goals were developed to be indicative of the thrust and contribu- 
tion which the Evaluation Department can make to the improvement of education 
in Columbus. These goals have been translated into areas of service. The 
areas of service which specifically relate to evaluation are as follows: 

I. Evaluation and accountability . — This area of service can be broken into 
three basic activities. One activity is the development of evaluation 
and accountability systems. The evaluation office has the responsibility 
for working with groups of decision-makers and other affected personnel 
to develop systems which will permit the evaluation of, and account- 
ability for specific functions and activities of a department, division 
or program. One such example is the development of a teacher appraisal 
system which has been put into use by the school system and has under- 
gone one year of operation at this point. Another such system was an 
accountability system for evaluating building environment. Teachers, on 
a building-by-building basis, annually respond to an instrument to 
assess the professional environment of their building on several areas. 
The responsibility of the evaluation officer in developing these systems 
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is to utilize evaluation technology in the development of the system, 
pilot test the system which has been developed, iron out the bugs, de- 
velop manuals or other training materials, and turn over the system to 
an appropriate manager or administrator. The evaluation office then 
has the responsibility for monitoring the operation of the system and 
for revising the system as necessary. 

A second major area is that of context evaluation. For purposes 
of convenience, context evaluation has been broken into five major areas. 
Those areas include achievement data, information on staff, program 
information, surveys, and community involvement procedures. 

The third activity of the evaluation and accountability section 
deals with ? nput, process, and product evaluation — those ad hoc evalu- 
ation activities which take place after a problem has been identified, 
a priority assigned to its solution, and action taken to design a solu- 
tion, implement it, and test it. 

2. Compensatory education service. — A second major service area is that 

of compensatory education service, which is to provide process and prod- 
uct evaluation service to funded projects such as Title I, ESEA and our 
State Disadvantaged Pupil programs, as well as Model Cities projects 
and other funded projects, including Title III programs. The Compensa- 
tory Education Service Unit utilizes the context data available to them 
from the evaluation and accountability section. 

3. Field Services Unit. — A third major area of services is that of Field 
Services. The concept of the Field Services Unit developed from the 
need to be able to respond to problems at the local building level. 
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The primary functions of the Field Services Unit are to deliver the 
expertise of f'e evaluation office to the field, to provide training 
and consultating services as well as limited, direct service to the 
faculty, principal, parents and students of a local school.. One major 
product that we have worked on in this area has been the development 
of a process for the evaluation of elementary schools on a school-by- 
school basis involving faculty, parents, and students. This grew out 
of the recently established State Minimum Standards for Elementary 
Schools in Ohio, and the need to develop a practical evaluation instru- 
ment and procedure which could be used and be useful. 

* 

Delineating, Obtaining 
and Providing Information 

As you may have noted in the definition of evaluation given previously, 
the three major activities of delineating, obtaining, and providing are 
performed by the evaluator in relation to decision-makers in the school 
system. The evaluator has the responsibility for delineating information 
needs with school system decision-makers from board members to teachers, for 
delineating information needs of the community and for providing a meaning- 
ful interface between these many and disparate needs. The evaluation office 
then performs the function of obtaining the information, from extant records, 
or by collecting new information when necessary, putting the information 
into meaningful form which relates to the decisions to be made, and finally, 
reporting this information to the appropriate audiences. Our decision-makers 
include students, parents, teachers, building principals, board members, etc. 
In the case of federally funded projects or state funded projects, our 
decision-makers include the State Department of Education and various U.S. 
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Office of Education departments or divisions. It has become apparent that 
not all decision-makers require the same information nor do they require 
information in the same form because decisions at different levels are dif- 
ferent. However, it is important to our school system that funding sources 
receive information about programs and projects which have been funded. It 
is also important to local decision-makers, whether they be program directors 
or community people, to have information which may not be the same as that 
which has been provided through the typical reporting system to the State 
Department or the U.S. Office. In fact, in most cases the decisions made 
at the local level in the community, in the school, and in the program may 
be more important to the life of the program and to the success of the pro- 
gram than those decisions which are made at other levels. Consequently, it 
is necessary to provide a level of support which will enable the evaluation 
office to be responsive to local decision-making information means, as well 
as to meet the mandated reporting requirements of funding sources. 

Problems Departments of Evaluation Incur 

Some basic problem areas of evaluation offices are: credibility, 

threat, personnel and limited resources. A discussion of each of these prob- 
lem areas fol lows with suggested remedies to each. 

Credib? 1 1 ty . — Because it must be responsive to many audiences, the 
evaluation office must develop and maintain credibility with each of these 
audiences. The evaluation office within a school system faces the problem 
of being too closely identified with any one group in the system. For 
example, it could be seen as an arm of the admin istrator, whether they are 
in the field or central office. In other systems, it could be seen as an 
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arm of the teachers. It is important to dispel these notions and maintain 
the image of being responsive to all audiences in a credible manner. How 
is this to be done? First, and perhaps most important, evaluation reports 
must be objective. Second, the process used for evaluation must be open to 
scrutiny — must be public by all parties concerned. One route being promoted 
to establish credibility of evaluation activities is the use of external 
auditors. Certainly this also has potential for establishing credibility. 

The evaluation office can also be the holder of the "good housekeeping seal 
of approval" for evaluations performed by other departments, schools or pro- 
grams in the school system. In other words, within the system, the office 
could serve as the external auditor for other evaluations. 

Threat. — This simple word represents a major concern which the word 
eva I uate genera 1 1 y raises among professionals, teachers and administrators 
alike. Establishing objectivity and demonstrating fairness and equity to 
all who are involved in evaluation is one sure way of removing the threaten- 
ing connotation of evaluation. It has been suggested that a "bill of rights" 
be established — to cover the rights of both the evaluator and the evaluatee. 

Person ne I . — The evaluation office must have competent personnel. It 
is important to recruit, train and maintain the personnel necessary to per- 
form a task at hand. I’d like to make a case for evaluation and account- 
ability personnel as being different. They are not classroom teachers on 
special assignment. They are not guidance counselors doing program evalua- 
tion and they are not principals who have retired from the battleground. 

They are competent specialists rather than generalists. They have probably 
received some specialized training or should receive such training to meet 
the information-providing needs of their position. More evaluators are now 
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being trained in tne field, on-the-job, than in our graduate schools. 
Universities should be doing more to train evaluation personnel to work in 
the field. This could easily be a collaborative venture of universities, 
school systems and other interested agencies. It is essential to foster 
these collaborative relationships. There are evaluation activities and 
developmental tasks which are beyond the means of any of these institutions 
operating alone. Through a device such as we have used — the internship — 
personnel receive, in addition to their formal classroom training, on-the- 
job training and a reality orientation; and we receive a source of personnel, 
a recruiting mechanism and a major source of ideas. 

Limited resources. — The most serious problem is that the evaluation 
office must have budgetary support for systematic evaluation. Information 
costs money — Up to one percent of total system budget has been recommended, 
similar to the R&D budgets of industry. The National School Boards Asso- 
ciation has recommended that up to two percent of school system budgets be 
allocated to evaluation, research and development. The evaluation of 
special funded projects generally is more costly than the systematic evalua- 
tion performed for the system. A rule of thumb has been used in projects 
such as Title I for a range of 5 to 7 percent of the total project budget. 

I think it is important to stress that evaluation costs are also opportunity 
costs. You may save thousands of dollars over several years by having 
evaluative information about .programs . You may avoid costly program invest- 
ments with low payoffs, or be able to terminate or modify such programs be- 
fore they become costly mistakes. We may then be able to maintain the 
public’s confidence that a course of action being taken is the best of 
available strategies. Unfortunately, there is never enough money available 
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to satisfy the evaluation needs of decision-makers. Consequently, trade- 
offs must be made. One such trade-off might be the funded project which 
requires completion of forms with census counts and other related informa- 
tion, readily obtainable from a context data base operated by the evalua- 
tion office for the total school system. If this information is available, 
then it becomes possible for the evaluation personnel or the evaluation 
budget available to the project to be concentrated on other forms of evalua- 
tion — a more sophisticated process for moni tori ng evaluation, a more complete 
product evaluation which will in turn give local decision-makers more 
information and enable them to have a higher probability of success in the 
project attaining its goals. Unfortunately, funding sources often fail to 
see the wisdom in supporting evaluation which is not directly responding to 
the questions on their forms. Although we consider it necessary to meet the 
mandated evaluation requirements, such as completing state or federal level 
forms, we also find it essential to provide useful information to local 
decision-makers concerning the project — information which can be fed back to 
the community, and to ail of the participants in the interest of improving 
the project and the outcomes of the project. 

Necessary Level of Funding 

Typically, state and federal government guidelines for specially funded 
programs or assisted projects call for an evaluation. However, guidelines 
fail to specify allowable amounts which can be expended upon evaluation, 
nor do the guidelines specify adequately the type of evaluation to be per- 
formed. Perhaps these shortcomings are due to the inability of evaluators 
themselves to specify general rules of thumb for establishing allowable 
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amounts of money required to evaluate and to specify the type of evaluation 
necessary in a certain situation. 

What resolutions are possible in this dilemma? First, it is essential 
that school systems provide budgets for evaluation, particularly to maintain 
the context data base which wi I I service the needs of the system and provide 
basic information for use by funded projects — and to place some value — 
through investment — on its activities. Secondly, it is necessary to identify 
for each project that information which must be provided to the funding 
source, whether it be state department or federal government — along with the 
cost of providing that information, whether that cost be specified in terms 
of full-time equivalent man-days or specified in terms of dollars. Then 
the means for providing information on the questions of local decision- 
makers, whether they be community people, board members, educators or other 
audiences, must be specified along with the cost of obtaining that informa- 
tion. The decision-maker, the people who have the responsibility and 
authority to allocate resources within the project, must then decide what 
information is necessary and assign priorities to that which has the greatest 
need. I believe that it is not only difficult but foolhardy at this point 
in the history of evaluation to begin to specify upper limits or even lower 
limits in terms of percentages of program budget which should be allocated 
to evaluation. It is more important to specify the information needs which 
are essential and important to the decision-makers, the decision-makers at 
the various levels: local, state and federal. There are small projects 

which, if carefully evaluated, have implications and potentials for applica- 
tion in many different settings. Obviously, this type of situation may well 
require a much higher percentage of the budget to be expended upon evaluation 
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than a situation where a large project is operating if it has operated for 
a number of years and most of the important questions have been answered. 

A manufacturing analogy might be appropriate in this situation. A new 
product of an R&D enterprise would be very carefully studied upon many 
criterion variables. For example, product shelf life may be very important. 
Product stability, product life under normal use, and product failure are 
all important aspects of tests which would be run in a laboratory and in 
reality settings on new products. Once this information has been established 
and adjustments have been made in the product and in the manufacturing pro- 
cess, then evaluation becomes a matter of quality control or product fidel- 
ity, such that we are sure we are getting the same product under the same 
conditions every time. This is to say that a new project on a start-up 
operation will require different kinds of evaluation to provide different 
kinds of information to the decision-makers, and that once a project reaches 
a point of stability and desirability, the matter of quality control or pro- 
cess evaluation then becomes important, to be accomplished by monitoring the 
process. 

This discussion has centered on the purpose and problems of internal 
departments of evaluation. It should be evident that internal departments 
can meet the needs of funding agencies as well as service to local education 
agency decision-makers. They have the added advantage of providing for con- 
tinuity and building a cumulative asset in the local agency. Proponents of 
external evaluation attack the competence of local education agencies to per- 
form adequate eva I uat ions. However, with the same specified levels of support, 
and with specifications for performance, LEA evaluation offices can be as 
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competent or more competent than the external agency. Systematic starva- 
tion for resources will create a self-fulfilling prophecy. 

It is proposed that funding agencies, in addition to requiring evalua- 
tion and information also provide that adequate funds be made available to 
support the costs of evaluation and information. It is further proposed 
that the U.S. Office of Education support the development of evaluation 
capabilities in local education agencies for the evaluation of local educa- 
tion agency programs and their attendant information needs. Such support 
can and should be tied into proposed NIE activities, as well as federally 
supported projects, such as Title I, ESEA. 

This proposal would strengthen LEA competencies, avoid the creation of 
another institution — the external evaluator — and meet the needs of funding 
agency decision-makers and LEA decision-makers. This proposal would enable 
us to provide for something to be tested in the LEA rather than discarding 
it without an adequate test of its viability. 
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